Image sensors, also known as imagers, were developed in the late 1960s and early 1970s primarily for television image acquisition, transmission, and display. An imager absorbs incident radiation of a particular wavelength (such as optical photons, x-rays, or the like) and generates an electrical signal corresponding to the absorbed radiation. There are a number of different types of semiconductor-based imagers, including charge coupled devices (CCDs), photodiode arrays, charge injection devices (CIDs), hybrid focal plan arrays, and CMOS imagers.
These imagers typically consist of an array of pixels containing photosensors, where each pixel produces a signal corresponding to the intensity of light impinging on that element when an image is focused on the array. These signals may then be stored, for example, to display a corresponding image on a monitor or otherwise be used to provide information about the optical image. The photosensors are typically phototransistors, photoconductors or photodiodes. The magnitude of the signal produced by each pixel, therefore, is proportional to the amount of light impinging on the photosensor.
To allow the photosensors to capture a color image, the photosensors must be able to separately detect, for example when using a Bayer pattern, red (R) photons, green (G) photons and blue (B) photons. Accordingly, each pixel must be sensitive only to one color or spectral band. For this, a color filter array (CFA) is typically placed in front of the pixels so that each pixel measures the light of the color of its associated filter. Thus, each pixel of a color image sensor is covered with either a red, green or blue filter, according to a specific pattern (e.g., Bayer pattern).
For most low cost CMOS or CCD image sensors, the color filters are integrated with the sensors. A common example of a color filter pattern is the tiled color filter array illustrated in U.S. Pat. No. 3,971,065, (the disclosure of which is incorporated by reference herein) and commonly referred to as “the Bayer pattern” color filter.
As shown in FIG. 1, the Bayer pattern 100 is an array of repeating red (R), green (G), and blue (B) filters causing pixels underneath the filters to be red, blue, and green pixels, respectively. In the Bayer pattern 100, red, green and blue pixels are arranged so that alternating red and green pixels are on a first row 105 of an array, and alternating blue and green pixels are on a next row 110. These alternating rows are repeated throughout the array. Thus, when the image sensor is read out, line by line, the pixel sequence for the first line reads GRGRGR etc., and then the alternate line sequence reads BGBGBG etc. This output is called sequential RGB or sRGB.
In the Bayer pattern 100, sampling rates for all three basic colors are adjusted according to the acuity of the human visual system. That is, green color, to which the human eye is most sensitive and responsive, is sensed with a larger number of sensors, whereas blue and red color, for which the human vision has less resolution, are sensed with a fewer number of sensors. This is why in the Bayer pattern, the green-sensitive elements occur at every other array position, while the red-sensitive elements and the blue-sensitive elements occur at every fourth array position.
As shown in FIG. 2 in a solid state image sensor, the Bayer patterned filters may be formed over an array 200 of pixel sensors 205. Specifically, an array 200 of pixel sensors 205 is formed on a semiconductor substrate 210. Each pixel sensor 205 has a photosensitive element 215, which may be any photon-to-charge converting device, such as a photogate, photoconductor or photodiode. The color filter array 220 may be formed over an upper level metal layer 225 in the array 200, separated from the photosensors 215 by various metallization and insulating layers such as an interlevel dielectric layer (ILD) 235 containing insulator metallization patterns and a passivation layer 230. The metal layer 225 may be opaque and used to shield the areas of the pixels that are not light sensitive. Convex lenses 240 are formed over the color filters 220. In operation, incident light is focused by the lenses 240 through the filters 220 to the photosensitive element 215.
For a Bayer pattern filter, values for red, green and blue are necessary for each pixel. Since each pixel sensor cell is only sensing one color, values for the remaining two colors are calculated by interpolation from the neighboring pixels that are sensing the missing colors. This color plane interpolation is known as demosaicing. For example, with reference to FIG. 1, pixel sensor cell 115 is associated with a green filter, which causes pixel sensor cell 115 to sense green light and produce a signal which represents only green light. In order to obtain an approximation of the amount of red and blue light for pixel sensor cell 115, a value may be interpolated from the neighboring red pixel sensor cells 120 and 125 and the neighboring blue pixel sensor cells 130 and 135, respectively. If demosaicing is not performed correctly, the resulting image may suffer from the inclusion of highly visible color artifacts.
The article entitled “Color Plane Interpolation Using Alternating Projections” published in the IEEE Transactions on Image Processing, Vol. II, No. 9 in September 2002 and written by Bahadir K. Gunturk, Yucel Altunbasak and Russell M. Mersereau (the disclosure of which is incorporated by reference herein) compares several demosaicing techniques. As described each of these demosaicing techniques have their respective advantages and disadvantages.
As described, for each pixel, the value used for a first color is based on a sensed color (i.e., the light sensed by the pixel's sensor 205) and the values for the remaining two colors are based on an interpolated value from the sensed values of corresponding neighboring pixels. Each of the sensed values are representative of the color value at the center of the pixel. Each of the interpolated color values are also representative of the value at the center of the pixel. The interpolated signal is inherently of a lower quality than the originally sensed signal. For example, an interpolated red color value at the center of a pixel would be different than a sensed red color value for the center of the same pixel.
Demosaicing methods, which typically only reconstruct the missing color components, may result in artifacts such as so-called zipper effects and random color dots. Additionally, there exists a trade off between filtering for noise and the blurring of sharp edges. Interpolating a pixel using the surrounding pixel values tends to blur the image when pixels at an edge are interpolated. Furthermore, this also leads to a decrease in the sharpness of the image at the edges. This is mainly due to the averaging of pixels which are across an edge contained within the image. The demosaicing method averages pixels so that a pixel at an edge does not have the sharpness desired of an edge pixel. Accordingly, there is a desire and need for color plane interpolation that provides improved demosaicing techniques that detect and account for image edges when calculating interpolation values.